Density Networks for Dimension Reduction of Continuous Data: Analytical Solutions∗

MacKay’s density networks (1995) provide a framework for generative modelling which can be adapted to specific problems by conveniently selecting a prior in latent space, a noise model in data space and a smooth, parametric mapping from latent to data space. The particular model thus obtained expresses the structure of a distribution in a high-dimensional data space in terms of a small number of variables in latent space. Here, we consider the problem of dimension reduction of continuous, high-dimensional data by density networks. We explore the possibilities of obtaining analytical expressions of the optimal parameters for general classes of distributions and mappings under maximum likelihood of the data. Only the case of a linear mapping and Gaussian prior and noise model seems to admit a straightforward solution, equivalent to factor analysis (and principal factor analysis). Nonlinear mappings and other families of distributions are not analytically tractable; approximate methods (e.g. Monte Carlo) are necessary in such situations, but then the complexity of the procedure becomes exponential on the number of latent variables due to the curse of the dimensionality.